Control system for a mobile manipulation device

ABSTRACT

A display window control system for a mobile manipulation device includes an interface and a processor. The interface is configured to receive an image from a mobile manipulation device. The processor is configured to determine a display output to display the image in a display window; determine an overlay output to display one or more overlay command icons; determine that an indication has been received to activate an overlay command; and provide the indication to the mobile manipulation device.

CROSS REFERENCE TO OTHER APPLICATIONS

This application also claims priority to U.S. Provisional PatentApplication No. 62/473,778 entitled LOW COST GENERAL-PURPOSE MOBILEMANIPULATOR FOR INDOOR USE filed Mar. 20, 2017 which is incorporatedherein by reference for all purposes. This application also claimspriority to U.S. Provisional Patent Application No. 62/474,427 entitledLOW COST GENERAL-PURPOSE MOBILE MANIPULATOR FOR INDOOR USE filed Mar.21, 2017 which is incorporated herein by reference for all purposes.This application also claims priority to U.S. Provisional PatentApplication No. 62/473,778 entitled ADDITIONAL HARDWARE AND SOFTWARE FORA LOW-COST GENERAL-PURPOSE MOBILE MANIPULATOR FOR INDOOR USE filed Feb.5, 2018 which is incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

People with limited abilities often need an assistant for performingtasks around the home. For example, moving objects, feeding pets,cleaning up, etc., become very difficult tasks for people withdifficulty walking around their home or picking up and carrying objects.Even for the able-bodied it would often be preferable to not have toperform some of these tasks, for example, when traveling, busy withwork, relaxing, etc. Hiring an assistant to come into the home isexpensive and can bring complications, for example introducing thepossibilities of theft, employment law issues, etc. However, the worldof humans is very messy and difficult for robots to navigate. Only thesimplest home robots have shown themselves to be practical (e.g.,robotic vacuum cleaners). This creates a problem where robots for morecomplicated tasks typically require very sophisticated technology,becoming more expensive than hiring a human assistant.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating an embodiment of a network systemfor mobile manipulation device control.

FIG. 2 is a diagram illustrating an embodiment of a mobile manipulationdevice.

FIG. 3 is a diagram illustrating an embodiment of a base.

FIG. 4 is a diagram illustrating an embodiment of a belt-driven linearactuator.

FIG. 5 is a diagram illustrating an embodiment of a manipulator end.

FIG. 6 is a diagram illustrating an embodiment of a manipulator end.

FIG. 7 is a diagram illustrating an embodiment of a manipulator end.

FIG. 8 is a diagram illustrating an embodiment of an arm actuator.

FIG. 9 is a diagram illustrating an embodiment of a mobile manipulationdevice in a room.

FIG. 10 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device.

FIG. 11 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device.

FIG. 12 is a diagram illustrating a set of command icon regions.

FIG. 13 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device.

FIG. 14 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device.

FIG. 15 is a flow diagram illustrating an embodiment of a process for adisplay window control system for a mobile manipulation device.

FIG. 16 is a flow diagram illustrating an embodiment of a process fordetermining a display output to display the image in a display window.

FIG. 17 is a flow diagram illustrating an embodiment of a process fordetermining an overlay output to display a coarse overlay command iconor a fine overlay command icon.

FIG. 18 is a flow diagram illustrating an embodiment of a process fordetermining an overlay output to display a coarse overlay command iconor a fine overlay command icon.

FIG. 19 is a flow diagram illustrating an embodiment of a process for aremote control interface system for a mobile manipulation device.

FIG. 20 is a flow diagram illustrating an embodiment of a process fordetermining actuator commands based at least in part on an indication tomove.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

A display window control system is disclosed. The display window controlsystem for a mobile manipulation device comprises an interfaceconfigured to receive an image from a mobile manipulation device and aprocessor configured to determine a display output to display the imagein a display window, determine an overlay output to display one or moreoverlay command icons, determine that an indication has been received toactivate an overlay command, and provide the indication to the mobilemanipulation device. In some embodiments, the display window controlsystem for the mobile manipulation device comprises a memory coupled tothe processor and configured to provide the processor with instructions.

A mobile manipulation device is disclosed. The mobile manipulationdevice comprises a base, a lift, an arm, and a manipulator. The base isable to move across a surface underneath the base. The lift is coupledto the base. The lift moves the arm vertically. The arm moves themanipulator horizontally along one direction. The base is able to moveperpendicular to the one direction.

A mobile manipulation device comprises a base, a lift, an arm, and amanipulator. The base moves the lift across a surface using actuatorsconnected to mechanisms such as wheels, tank treads, or articulatedlinks. The base can rotate and translate the lift across the surface andin some embodiments perform other motions. In some embodiments, the baseuses two driven wheels. The lift moves the arm vertically—lifting andlowering. The arm moves horizontally—extending and retracting. The basecan move perpendicular to the arm's horizontal motion. The manipulatoris attached to the arm. Together, the base, lift, and arm result inCartesian motion of the manipulator. The base is also responsible formoving the lift to different locations and orientations in theenvironment.

People with limited mobility, cognitive impairment, perceptualdifficulties, or reduced dexterity would benefit from a home robot thatretrieves requested objects and performs other manipulation tasks. Herea robot is described that can perform manipulation tasks in humanenvironments, such as homes. The robot emulates advantageouscharacteristics of the human body in a low-cost and highly-simplifiedform.

A full humanoid robot would be well adapted to manipulating everydayobjects in human environments, however the cost would be prohibitive andthe robot would be challenging to control. There are several importantcharacteristics of the human body such a robot should have, includingsensors and manipulators that are high above the ground, a smallfootprint, and stability with respect to perturbations, even whenreaching out into the environment in order to perform tasks, such asgrasping objects. The disclosed device is a robot that emulates theseand other humanoid characteristics in a low cost, simplified form.

Two key capabilities of the design are its ability to reach importantlocations in human environments and its ability to take advantage of theCartesian structure of human environments, which tend to consist of manyhorizontal and vertical planes, such as floors, tables, countertops, andcabinets. Some of the novel concepts included follow:

-   -   Use of fewer actuators by using the base during manipulation to        achieve movement along an additional orthogonal direction; use        of a thin and long telescoping mechanism to reach through        clutter instead of the typical approach of reaching around        clutter with multiple rotary joints; and use of a Cartesian        structure matched to the Cartesian structure of indoor human        environments to simplify manipulation.    -   Use of smaller actuators by having the arm actuator move a        telescoping member orthogonal to gravity to avoid issues of        gravitational loading and long moment arms typically associated        with a long reach; using a telescoping member rigidly attached        to a carriage on the lift and the lift attached to a high-mass        mobile base to enable the robot's structure to support the force        and moment applied by the telescoping arm to the carriage        instead of an actuator; and using a lift actuator that moves a        low mass arm assembly up and down, where in some embodiments the        arm assembly is low mass due in part to the small arm actuator        and the lightweight telescoping structure.    -   Reduced need for coordination among actuators due to Cartesian        structure in Cartesian environments, which enables independent,        serialized control of the mobile base actuators, lift actuator,        and telescoping arm actuator to perform many useful tasks in        human environments. For example, when reaching to a location, a        useful strategy is to first position the base, then raise or        lower the arm so that the arm points at the target location, and        then extend the arm until the manipulator reaches the location.        Each of these motions can be facilitated by cameras and refined        with further motions.    -   Safe and stable reaching to locations due to a lightweight        telescoping arm that is long and has a small cross section to        reach through clutter; a statically stable base with a small        footprint to navigate to locations in human environments; and a        low center of mass resulting from a relatively large mass in the        base from components such as the batteries and wheel actuators        and a lightweight structure above the base due to the        lightweight lift, arm, and manipulator.    -   Versatility through specialized manipulators and tools that can        be manually or automatically attached to the end of arm. These        manipulators and tools may include simple or complex        manipulators and tools—for example, simple unactuated tools like        a hook, complex actuated mechanisms like a pitch-roll wrist,        and/or actuated tools like a vacuum. Other examples of        manipulators and tools include: a two actuator wrist with a        compliant gripper; a gripper at a fixed orientation; hooks for        opening/closing drawers and doors, operating light switches, and        operating elevator buttons; a spoon for feeding; an adjustable        mobile phone/tablet holder; a dustpan like gripper to pick        things up from the ground; a vacuum cleaner brush; a small brush        tool and dustpan attached to the arm to sweep debris off        surfaces; and tools to leave marks, such as a marker holder with        a spring to write on surfaces.    -   View of the world from a human perspective without a complex        robot head by using cameras with wide fields of view mounted at        human eye level.

The disclosed robot design uses a single long, narrow, lightweighttelescoping mechanism to reach locations around a person's body. Inconjunction with the robot's wheeled base and vertical lift, the robotcan directly move objects in orthogonal directions matched to the planarstructure of human environments, simplifying control of the robot. Thisdesign reduces the robot's overall weight, number of actuators, andactuator requirements. In addition the robot's motions are easier forpeople to understand and direct.

In some embodiments, this robot design uses four motors: two for thebase wheels, one for a long, narrow, lightweight telescoping mechanismthat extends horizontally and serves as the robot's arm, and one motorfor a vertical lift that raises and lowers the robot's telescoping arm.The robot's arm extends and retracts forward and backward, the liftmoves up and down, and the wheels move the robot sideways or rotate therobot. These four motors provide four degree of freedom control of theend of arm (X, Y, Z, and theta). Less than four motors can be used usinga clutched transmission and more than four motors can be used using aredundant mechanism. An actuated manipulator or a simple tool, such as ahook or duster, can be attached to the end of the arm. For example, thearm moves a manipulator toward and away from a base mounting pole alongan x-axis; the arm is moved up and down along a base mounting polerelative to the floor along a z-axis; and the base mounting pole thatprotrudes vertically from a base is moved perpendicular to the arm alonga y-axis. The base is also enabled to rotate around the axis of the basemounting pole. With these motions the end of the robot's arm can move infour directions using the telescoping arm, the lift, and the wheels.

The use of a long, narrow, lightweight, telescoping mechanism allows therobot to stably reach long distances relative to the footprint size ofits wheeled mobile base, and to do so with a relatively small motor. Byremaining horizontal, the motion of the arm does not work againstgravity when used on the flat ground of indoor human environments. Thelong reach is especially useful in cluttered human environments, such aswhen assisting a person by moving an object close to his or her body. Akey insight is that a long, thin telescoping arm with Cartesianpositioning and a single axis of rotation can reach most locations inhuman environments and thereby do useful things. In some embodiments,the telescoping arm has a cross section smaller than or similar in sizeto a cross section of a human arm and a length that is longer than orsimilar in length to an outstretched human arm. In some embodiments, thetelescoping arm is constructed from a series of nested structuralelements and sliding elements such as bushings or bearings and is anexoskeleton comprising a hollow structure that provides structuralsupport and also serves as the exterior of the arm.

The drive for the telescoping mechanism uses a large, somewhat stiffcable with electrically conducting wires to push and pull the end of thetelescoping mechanism while extending and retracting. The telescopingmechanism can be moved by placing the cable between a high-frictionpulley wheel attached to a motor and a passive pulley wheel with bothwheels compressed together with springs to maintain contact with thecable. The conducting wires carry power and signals to and from endeffectors at the end of the telescoping mechanism. The end effectors canbe changed, and the base of the robot can have a tool holder forenabling the robot to automatically change the end effectors. In variousembodiments, the end effectors include one or more of the following: anactuated grabber device, a dexterous hand, a robotic wrist, a vacuum, asuction gripper, a dustpan, a duster, a wiping element, a scraper, ahook, a rotary tool, a mop, a mobile phone holder, a tablet holder, abrush, a writing instrument, an eating utensil, a cloth, or any otherappropriate tool.

The disclosed mobile manipulator is an improved system because itprovides low cost, effective assistance with an intuitive controlmechanism as a human helper robot for human spaces.

FIG. 1 is a block diagram illustrating an embodiment of a network systemfor mobile manipulation device control. In the example shown, mobilemanipulation device control system 102 communicates with mobilemanipulation device 104 via network 100. Mobile manipulation devicecontrol system 102 comprises a computer, a smartphone, custom mobilemanipulation device control hardware, an analog mobile manipulationdevice controller, a digital mobile manipulation device controller, amobile manipulation device controller including a display, a manualmobile manipulation device controller, an automatic mobile manipulationdevice controller, a mobile manipulation device controller utilizingartificial intelligence, etc. Mobile manipulation device 104 comprises amobile manipulation device for performing tasks—for example, picking upobjects, carrying objects, cleaning floors or other surfaces, operatingmachinery, assisting humans, playing with pets, etc. Network 100comprises one or more of the following: a local area network, a widearea network, a wired network, a wireless network, the Internet, anintranet, a storage area network, or any other appropriate communicationnetwork. Mobile manipulation device 104 provides data from sensors(e.g., a top-mounted camera, an arm-mounted camera, a manipulatorcamera, a grip sensor, a position sensor, microphones, etc.) to mobilemanipulation device control system 102. The sensor data is displayed toa user of mobile manipulation device control system 102. A command isreceived from mobile manipulation device control system 102 and sent tomobile manipulation device 104. The command is translated to causeaction in mobile manipulation device 104 (e.g., motion of the arm,manipulator, base, etc.)

FIG. 2 is a diagram illustrating an embodiment of a mobile manipulationdevice. In some embodiments, mobile manipulation device 200 comprisesmobile manipulation device 104 of FIG. 1. In the example shown, mobilemanipulation device 200 comprises base 204, vertical lift 202, arm 216,and manipulator 222. Base 204 comprises a base for carrying and movingthe elements of mobile manipulation device 200 (e.g., along y axis orrotating around to change theta). Base 204 includes two or more wheels(e.g., wheel 206). The two or more wheels are driven. For example, thetwo or more wheels are driven together, independently, in common mode(e.g., in the same direction at the same speed), in differential mode(e.g., in opposite directions at the same speed), etc. In someembodiments, base 204 additionally includes an additional wheel, whereinthe additional wheel is not driven. For example, the additional wheelcomprises an additional wheel for providing static stability (e.g.,allowing the mobile manipulation device to balance without control). Theadditional wheel comprises a pivoting caster, a ball caster, a mecanumwheel, an omniwheel, etc. In some embodiments, base 204 additionallycomprises one or more additional wheels, wherein the additional wheelsare or are not driven. In various embodiments, base 204 additionallycomprises batteries, a computer, speakers, a vertical lift driver foractuating vertical lift 202, or any other appropriate elements.

Vertical lift 202 comprises a vertical lift for carrying, raising, andlowering arm 216 (e.g., along the z-axis). In some embodiments, verticallift 202 comprises a fixed mast (e.g., a vertical element of fixeddimensions) coupled to a linear actuator (e.g., an actuator for movingan element along a linear path), and arm 216 is coupled to the movingelement of the linear actuator. In various embodiments, the linearactuator comprises a belt drive, a lead screw, a ball screw, a linearactuator including a prismatic joint, etc. In some embodiments, verticallift 202 comprises a telescoping mast (e.g., a vertical element capableof extending or contracting in length), and arm 216 is coupled to afixed point on the telescoping mast. In the example shown, coupling 212comprises an actuated coupling for coupling arm 216 to vertical lift202. A linear actuator included in vertical lift 202 drives coupling 212and arm 216 up and down.

Camera mounting structure 208 is attached to the top of vertical lift202 and holds camera 210. Camera 210 comprises a camera for viewing theenvironment around mobile manipulation device 200. For example, camera210 comprises a camera facing the ground. When mounted high above theground, camera 210 can see the tops of surfaces in human environments,like countertops, tables, desks, and manipulable objects in placespeople commonly place them. In some embodiments, camera 210 comprises afisheye lens. When near human eye height with a fisheye lens, camera 210provides a view of the world comparable to a standing human. Cameramounting structure 208 for camera 210 can increase visibility of thesurrounding environment by reducing occlusions. Camera mountingstructure 208 places the camera away from the mast to reduce occlusionfrom the mast. In some embodiments, camera mounting structure 208 arcsbehind the camera to reduce occlusion, and its structure can usecomponents with thin cross sections oriented parallel to rays emanatingfrom the optical axis of the camera to reduce occlusion. Mobilemanipulation device 200 additionally comprises camera 214 mounted on arm216 so that it can view the arm and the manipulator. Camera 214 ismounted on arm 216 near vertical lift 202 and is facing gripper 222.Camera 214 comprises a camera for viewing the area where gripper 222 isoperating. In some embodiments, camera 214 comprises a fisheye lens.

Arm 216 comprises an arm extending horizontally from vertical lift 202(e.g., along x axis). In the example shown, arm 216 has a squarecross-section. In some embodiments, arm 216 has a round cross-section.Other cross sections can be used—both nest and non-nesting. In addition,keyed cross sections can be used that prevent rotation of thetelescoping tubes relative to one another. Arm 216 is telescoping—forexample, capable of a telescoping action for moving gripper 222 towardsor away from vertical lift 202. When the wheels of base 204 are drivenin common mode such that mobile manipulation device 200 moves in astraight line, mobile manipulation device 200 moves perpendicular to thedirection of arm 216. Arm 216 comprises hinge 218—for example, anactuated hinge for bending. In the example shown, when hinge 218 bends,gripper 222 moves toward the ground. Arm 216 additionally comprisescamera 220. Camera 220 is mounted on arm 216 near gripper 222. Asgripper 222 moves (e.g., due to the motion of hinge 218 or an actuatedrotational coupling), camera 220 moves with it holding gripper 222static in its frame. Camera 220 allows a close view of the actions ofgripper 222. In some embodiments, camera 220 comprises a fisheye lens.Gripper 222 is attached to the end of arm 216. In some embodiments, adifferent manipulator is attached to the end of arm 216 (e.g., amanipulator for interacting with other objects). For example,manipulator comprises a grabber, an interchangeable tool holder, avacuum, or a mop. Mobile manipulation device 200 may include a set ofinterchangeable tools that can be attached to an interchangeable toolholder manipulator. For example, a set of interchangeable tools can bemounted on base 204 and reached by retracting arm 216, folding hinge218, and lowering arm 216 using vertical lift 212. The interchangeabletool holder attaches to a variety of tools, including, but not limitedto, a grabber, a dexterous hand, a vacuum, a suction gripper, a dustpan,a duster, a wiping element, a scraper, a hook, a rotary tool, a mop, amobile phone holder, a tablet holder, a brush, a writing instrument, aneating utensil, and a cloth.

Arm 216 additionally comprises actuated rotational coupling 224 forrotating. Rotational coupling 224 makes grippers and other tools moreversatile. For example, it can be used to turn a door knob that agripper is holding. When hinge 218 makes the gripper point towards theground, rotational coupling 224 can orient a gripper to pick up anelongated object sitting on a flat surface from above. Rotationalcoupling 224 can also enable a hook to be oriented to hook onto bothvertical and horizontal drawer handles. Angle sensors on rotationalcoupling 224 can be used to rotate video from camera 220 to make it lookas though the camera was not being rotated by rotational coupling 224and simplify remote operation by a human.

Camera 210, camera 214, and camera 220 provide video data to aninterface and a processor disposed in base 204 of mobile manipulator200. The processor provides the video data to a mobile manipulationdevice control system. The mobile manipulation device control system maybe remotely located from the robot, allowing for remote teleoperation orautonomous control. The mobile manipulation device control systemprovides a user one or more views from the video data and receivesmotion commands from a user. The motion commands are provided to mobilemanipulator 200. An interface of mobile manipulator 200 (e.g., acommunication interface) receives the commands and provides the commandsto a processor of mobile manipulator 200 and are then translated tospecific motion commands for actuators of mobile manipulator 200 (e.g.,fine or coarse motions for each of base 204, vertical lift 202, arm 216,folding hinge 218, and manipulator 222). In some embodiments, lightrings or infrared sources are placed around some or all of the camerasto enable operating in dark rooms and other dark areas like the interiorof a container.

In some embodiments, a control system for a mobile manipulation devicecomprises an interface that enables a remote operator to drive themobile manipulation device and manipulate objects. The interface is avideo-centric interface. The operator directly clicks on or touches oneor more video streams to make the mobile manipulation device move. Forexample, clicking on or touching various parts of the video stream fromthe top camera makes the mobile manipulation device's wheeled base move.Clicking on or touching various parts of the video stream from the armcamera makes the arm move up and down and extend and retract. Clickingon or touching various parts of the video stream from the wrist cameramakes the gripper open and close, bend down, straighten up, roll right,and roll left.

The regions are positioned with respect to the mobile manipulationdevice to make the mapping between a click or touch and mobilemanipulation device motion intuitive. For example, clicking on ortouching the video above the arm makes it move up, and clicking ortouching in front of the arm makes it extend. The interface alsoprovides feedback on the grip force and motor torques by making relevantregions of the video stream turn red with darker, less transparent, redmeaning higher force or torque. The operator can use this to betterinfer what is happening. The red coloring is displayed in the regionthat when clicked or touched is likely to increase the torque or gripforce. Each region of a video stream corresponds with a command can havea distinctive cursor icon associated with it as well as a descriptivetext tooltip that appears when the cursor is held in the region for anextended duration. All commands are executed with a single click ortouch, which results in one of the robot's joints moving a predefineddistance/angle or executing a predefined autonomous motion of a shortand bounded time. Some clickable or touchable regions in the video moveover greater or smaller distances/angles. This is similar to a ‘jogmode’ on a Computer Numerical control (CNC) machine where coarse andfine motion of individual axes can be attained using a push buttoninterface. The short bounded time of the resulting motion in ‘jog mode’allows the robot operator to safely test small motions, observe in thevideo how they change the state of the world, and then adjust thesubsequent set of jog commands. It also allows the robot operator toremove their attention from the robot operation at any time withouthaving to consider returning to a robot that is an unsafe state.

In some embodiments, the interface rotates the gripper video to make itappear as though the camera is always in the same orientation withrespect to gravity (i.e., the ceiling is always at the top of the imageand the floor is always at the bottom) in spite of the camera rollingwith the gripper. In some embodiments, there is also a microphonemounted to the gripper that provides audio feedback to the operatorabout objects being manipulated and helps the operator hear what themobile manipulation device is doing, providing better situationalawareness.

In some embodiments of the interface, the interface comprises multipleuser selectable operator interface modes that map to primary usages ofthe robot. In some embodiments, a navigation mode, a manipulation mode,a grasp object from the top mode, and a grasp object from the side modeare provided. Each user interface mode presents one or more camera videostreams that are zoomed, cropped, translated, and otherwise modified topresent a very intuitive mapping between what the operator sees, whatthe operator clicks on or touches, and what the robot does to accomplishits task. By changing which videos are displayed, how the videos areprocessed, the mapping from locations on the videos to robot actuatorcommands, the computer graphics feedback provided to the user, and otheraspects of the interface, a user selected interface mode can make a taskmore intuitive to perform with the robot.

The navigation mode makes driving the robot through an environment moreintuitive. The navigation mode shows video from the top camera rotatedsuch that the top of the video display corresponds with the direction offorward motion of the mobile manipulation device (i.e., y axis isvertically aligned with the video display). Clicking on or touchinglocations in the top middle part of the display results in forwardmotion of the robot along they axis and clicking on or touchinglocations near the bottom middle of the display results in backwardmotion of the robot along the y axis. Clicking on or touching locationson the sides of the display results in rotation of the robot around thetheta axis. In some embodiments, the navigation mode simultaneouslyshows video from the arm camera to help the user avoid hitting the arminto something while driving the mobile manipulation device. FIG. 13shows an arm camera interface comparable to what is used by someembodiments of the navigation mode. In some embodiments, video issimultaneously shown from the wrist camera to provide additional contextwhile navigating the robot through an environment. FIG. 14 shows a wristcamera interface comparable to what is used by some embodiments of thenavigation mode. In some embodiments, the size of the top camera videodisplay is larger than other video displays to emphasize its importanceand provide greater detail from the top camera while driving the robot.

The manipulation mode makes manipulating objects with the robot moreintuitive. The manipulation mode shows video from the top camera rotatedsuch that the top of the video display is the direction in which the armextends (i.e., x axis is vertically aligned with the video display).FIGS. 10, 11, and 12 show a top camera interface comparable to what isused in some embodiments of the manipulation mode. In some embodiments,the manipulation mode can simultaneously show video from the wristcamera with an interface similar to the interface shown in FIG. 14. Insome embodiments, the manipulation mode also simultaneously shows videofrom the arm camera. FIG. 13 shows an arm camera interface comparable towhat is used by some embodiments of the manipulation mode. In someembodiments, the size of the wrist camera video display is larger thanother video displays to emphasize its importance and provide greaterdetail from the wrist camera while manipulating objects with the robot.

The grasp object from the top mode makes grasping objects from abovemore intuitive. The grasp object from the top mode is used when thegripper is pointing down toward the ground (i.e., perpendicular to thearm). The grasp object from the top mode displays video from the wristcamera. Clicking on or touching the video display results in the grippermoving across a flat horizontal plane in order to position the gripperabove an object to be grasped. This is achieved by sending commands tothe base (y axis) and the arm (x axis). Clicking on or touching the topor bottom of the video display results in motion of the arm (x axis).Clicking on or touching the sides of the video display results in motionof the base (y axis). Once positioned above an object, the gripper canbe rotated and the gripper can be lowered (z axis) and then closed tograsp the object.

The grasp object from the side mode makes grasping objects from the sidemore intuitive. The grasp object from the side mode is used when thegripper is pointing straight out (i.e., parallel to the arm). The graspobject from the side mode displays video from the wrist camera. Clickingon or touching the video display results in the gripper moving across aflat vertical plane in order to position the gripper in front of anobject to be grasped. This is achieved by sending commands to the base(y axis) and the lift (z axis). Clicking on or touching the top orbottom of the video display results in motion of the lift (z axis).Clicking on or touching the sides of the video display results in motionof the base (y axis). Once positioned in front of an object, the arm canbe extended (x axis) and the gripper closed to grasp the object.

By making the control system for the mobile manipulation systemintuitive and easy to use, control of the mobile manipulation systembecomes available to a wide range of non-specialized users. The controlsystem allows control of all degrees of freedom of the mobilemanipulation system with only a mouse and using a display with minimaldistractions. In some embodiments, a user can command the system bytouching the display rather than using a mouse. The system includes aplurality of cameras allowing the user to see different context formobile manipulator system motions and automatically determines a userinterface behavior based on the camera currently being displayed. Thiscontrol system for a mobile manipulation system significantly broadensthe ability of manipulator robots to perform useful work in the home.

FIG. 3 is a diagram illustrating an embodiment of a base. In someembodiments, base 300 comprises base 204 of FIG. 2 (e.g., a base of amobile manipulation device). In the example shown, base 300 includeswheel 304 and wheel 308. Wheel 304 and wheel 308 comprise wheels fordriving the mobile manipulation device. Wheel 304 and wheel 308 arecapable of driving the mobile manipulation device in a straight linewhen driven in common mode (e.g., in the same direction at the samerate) or of turning the mobile manipulation device when driven indifferential mode (e.g., in opposite directions at the same rate). Inthe example shown, wheel 304 is driven by driver 306 and wheel 308 isdriven by driver 310. In some embodiments, wheel 304 and wheel 306 aredriven by a single driver (e.g., including a switchable drive train fordriving wheel 304 and wheel 306 in common mode or in differential mode).Base 300 additionally comprises wheel 312. Wheel 312 comprises a thirdwheel that is not driven. In the example shown, wheel 312 comprises aball caster. In various embodiments, wheel 312 comprises a pivotingcaster, a mecanum wheel, an omniwheel, or any other appropriate wheeltype. Wheel 312 comprises a wheel for providing static stability. Insome embodiments, base 300 additionally comprises one or more additionalwheels, wherein the wheels are driven or not driven.

Base 300 additionally comprises computer 302. Computer 302 comprises acomputer including an interface system, a memory, a processor, datastorage, etc. Computer 302 communicates with a mobile manipulationdevice control system (e.g., mobile manipulation device control system102 of FIG. 1) via a network for providing data (e.g., camera data,microphone data, actuator data, sensor data, etc.) and for receivingcontrol information. Computer 302 comprises connections to other systemsof the mobile manipulation device, for example, a power connection tobattery 316, driver connections to driver 306 and driver 310, acontroller connection to vertical lift controller 314, an audio dataconnection to send audio data to speaker 318, an audio data connectionto send audio data from a microphone on the robot to the mobilemanipulation device control system, a video data connection to receivedata from one or more cameras, an arm controller to cause a manipulatorto move toward and away from a vertical lift, a hinge controller tocontrol a hinge motion bending a manipulator up or down, a manipulatorcontroller for opening or closing a manipulator, a manipulatorcontroller for rotating the manipulator, etc. Vertical lift controller314 comprises a vertical lift controller for holding and actuating avertical lift to move an arm up and down. Battery 316 comprises abattery pack for powering the mobile manipulation device. Speaker 318comprises a speaker for providing audio (e.g., providing audio feedbackto a user of the mobile manipulation device) and in some implementationsenabling a remote human operator or artificial intelligence tocommunicate with people near the robot.

FIG. 4 is a diagram illustrating an embodiment of a belt-driven linearactuator. In some embodiments, the belt driven linear actuator of FIG. 4comprises vertical lift 202 and coupling 212 of FIG. 2. In the exampleshown, mast 400 comprises a vertical mast for holding the linearactuator. Rail 402 is fastened to mast 400. Rail 402 comprises atrapezoidal shape. Slider 404 comprises a slider for sliding linearly onrail 402. Slider 404 comprises a trapezoidal gap that fits thetrapezoidal shape of rail 402. The trapezoidal coupling prevents slider404 from moving in any direction other than up and down. Together,slider 404 and rail 402 form a prismatic joint. Belt 406 drives slider404 up and down. Belt 406 is driven by a vertical lift actuator (e.g., avertical lift actuator mounted in a mobile manipulation device base).Arm 410 is mounted to slider 404 and extends horizontally. In theexample shown, arm 410 has a round cross-section. Camera 408 is mountedto arm 410 via slider 404 and offset form the arm in order to see arm410 and a manipulator from the side.

In some embodiments, a compliant grabber device comprises two compliantfingers each with a suction cup fingertip. In some embodiments, eachcompliant finger has a compliant linkage that comprises two strips ofmaterial that behave as springs with one strip of material rigidlyaffixed to a housing and the other strip of material moved via anactuator to bend the finger. In some embodiments, the actuator pulls ona cable attached to the compliant linkages in order to cause the fingersto close and the return force of the springs causes the fingers to openwhen the cable tension is released. In some embodiments, a linearactuator rigidly attached to the compliant linkages pulls on thecompliant linkages to cause the fingers to close and pushes on thecompliant linkages to cause the fingers to open. In some embodiments,bend sensors mounted on the fingers provide signals with informationabout the kinematics and dynamics of the compliant grabber device. Insome embodiments, an actuator used to move the fingers provides voltage,current, and kinematic signals with information about the kinematics anddynamics of the compliant grabber device.

FIG. 5 is a diagram illustrating an embodiment of a manipulator end. Arm500 comprises an arm of a mobile manipulation device. In someembodiments, arm 500 comprises arm 216 of FIG. 2. In the example shown,arm 500 has a square cross-section. Arm 500 comprises rotational coupler502. Rotational coupler 502 comprises an actuated rotational couplingfor rotating. Arm 500 additionally comprises hinge 504 including hingeactuator 506. Hinge 504 comprises an actuated hinge for bending. Arm 500additionally comprises arm end 508. Arm end 508 rotates when actuated byrotational coupler 502 and swings when actuated by hinge 504. Camera 510is mounted on arm end 508. Camera 510 includes ring light 534 toilluminate area in front of camera 510.

Arm end 508 additionally comprises spring 512 and spring 514, spring 516and spring 518, and grabber 520 and grabber 522. Spring 516 and spring518 comprise spring elements capable of being actuated by being drawninto the end of arm end 508. In some embodiments, spring 516 and spring518 are connected to a cable pulled and released by an actuator. In someembodiments, an actuator is rigidly coupled to spring 516 and spring 518to push and pull them. In the example shown, linear actuator 532 isrigidly coupled to spring 516 and spring 518. When spring 516 and spring518 are drawn into the end of arm end 508, spring 512 and spring 514 arepulled together, and grabber 520 and grabber 522 are pulled together.Grabber 520 and grabber 522 can pick up an object in this way. When theactuation is reversed, spring 512 and spring 514 return to their defaultpositions or, in the case of a rigid coupling, can be pushed out to openbeyond their default positions. When spring 516 and spring 518 arepushed out of the end of arm end 508, spring 512 and spring 514 arepushed apart, and grabber 520 and grabber 522 are pushed apart. Grabber520 and grabber 522 can reach around a larger object than would fitbetween them in their default position in this way. Hook 530 attached tothe outside of grabber 522 can be used to hook onto drawer handles,light switches, door handles, and other objects and be used in othermanipulation tasks.

In various embodiments, hook 530 is rectangular, the protruding end ofan L-shaped attachment to the gripper, or any other appropriate shape.

Arm end 508 additionally comprises laser light source 524 and laserlight source 526. For example, laser light source 524 and laser lightsource 526 comprise laser pointers. Laser light source 524 and laserlight source 526 are positioned at a slight angle to arm end 508 suchthat the light from laser light source 524 converges with the light fromlaser light source 526 between grabber 520 and grabber 522 (e.g.,converging on a point indicating where the manipulator will interactwhen actuated). When the manipulator is being maneuvered into a positionto grab an object using grabber 520 and grabber 522, light from lightsource 524 and laser light source 526 reflecting from the object can beused to judge where the object is relative to grabber 520 and grabber522 (e.g., when viewing images captured by camera 510).

Bend sensors (e.g., bend sensor 528 on spring 514) mounted on spring 512and spring 514 provide information about the state of the gripper. Theactuator coupled to spring 516 and spring 518 provides current, voltage,and kinematic information about the state of the gripper. For example,together, this information can be used to detect contact, estimate theapplied grip force, estimate the width of the opening of the gripperwhen grasping an object, and perform other sensing.

Coupler 534 is used to quickly attach or decouple different tools fromthe end of arm 500.

FIG. 6 is a diagram illustrating an embodiment of a manipulator end. Arm600 comprises an arm of a mobile manipulation device. In someembodiments, arm 600 comprises arm 216 of FIG. 2. In the example shown,arm 600 has a square cross-section. Arm 600 additionally comprises armend 604. Camera 606 is mounted on arm end 604. Camera 606 includes ringlight 608 to illuminate area in front of camera 606.

Arm end 604 additionally comprises springs (e.g., spring 610) andgrabbers (e.g., springs and grabbers similar to FIG. 5). The springelements are capable of being actuated by being drawn into the end ofarm end 604. In some embodiments, springs are connected to a cablepulled and released by an actuator. In some embodiments, an actuator isrigidly coupled to springs to push and pull them. When springs are drawninto the end of arm end 604, springs are pulled together, and grabbersare pulled together. Grabbers can pick up an object in this way. Whenthe actuation is reversed, springs return to their default positions or,in the case of a rigid coupling, can be pushed out to open beyond theirdefault positions. When springs are pushed out of the end of arm 604,springs are pushed apart, and grabbers are pushed apart. Grabbers canreach around a larger object than would fit between them in theirdefault position in this way. Hook 618 attached to the outside ofgrabber can be used to hook onto drawer handles, light switches, doorhandles, and other objects and be used in other manipulation tasks.

Arm end 604 additionally comprises laser light source 614 and laserlight source 616. For example, laser light source 614 and laser lightsource 616 comprise laser pointers. Laser light source 614 and laserlight source 616 are positioned at a slight angle to arm end 604 suchthat the light from laser light source 614 converges with the light fromlaser light source 616 between grabbers (e.g., converging on a pointindicating where the manipulator will interact when actuated). When themanipulator is being maneuvered into a position to grab an object usinggrabbers, light from light source 614 and laser light source 616reflecting from the object can be used to judge where the object isrelative to grabbers (e.g., when viewing images captured by camera 606).

Bend sensors (e.g., bend sensor 612 on spring 610) mounted on springsprovide information about the state of the gripper. The actuator coupledto springs provides current, voltage, and kinematic information aboutthe state of the gripper. For example, together, this information can beused to detect contact, estimate the applied grip force, estimate thewidth of the opening of the gripper when grasping an object, and performother sensing.

Coupler 602 is used to quickly attach or decouple different tools fromthe end of arm 600.

FIG. 7 is a diagram illustrating an embodiment of a manipulator end. Insome embodiments, arm 700 comprises arm 216 of FIG. 2. In the exampleshown, arm 700 includes an actuator internal to arm 700 able to extendand retract coupler 702. When coupler 702 is retracted outer spring 704and outer spring 708 move grippers inward (e.g., gripper 706). Whencoupler 702 is extended outer spring 704 and outer sprint 708 movegrippers outward (e.g., gripper 706).

FIG. 8 is a diagram illustrating an embodiment of an arm actuator. Insome embodiments, arm 800 is used to implement x axis motion of arm 216.In the example shown, arm 800 is actuated using stiff but flexible cable808 and causes arm to extend and retract. Actuator 804 is for movingdrive pulley wheel 802. In some embodiments, cable includes power andsignal conductors used by a tool at the end of the arm. Unactuatedpulley wheel 806 and actuated pulley wheel 802 are held together withcompliant springs to move stiff cable 808 (e.g., a friction drivepulley). Hollow nested elements of arm 800 enable the end of arm 800 toextend and retract.

In some embodiments, the telescoping arm has a cross section smallerthan or similar in size to a cross section of a human arm. The smallcross section, long reach, and low mass of the telescoping arm areimportant features of the actual invention, since they enable the arm tostably reach places in human environments. In some embodiments, thetelescoping arm has a length that is longer than or similar in length toan outstretched human arm. In some embodiments, the telescoping arm isconstructed from a series of nested structural elements and slidingelements such as bushings or bearings. In some embodiments, thetelescoping arm is an exoskeleton comprising a hollow structure thatprovides structural support and also serves as the exterior of the arm.In some embodiments, the telescoping mechanism is driven by an elementthat contains power and signal conductors. In some embodiments, thepower and signal conductors in the cable used to extend or retract thearm are used by the manipulator or tool attached to the end of thetelescoping arm.

FIG. 9 is a diagram illustrating an embodiment of a mobile manipulationdevice in a room. In some embodiments, mobile manipulation device 900comprises mobile manipulation device 200 of FIG. 2 located in a room—forexample, a living room of an apartment. In the example shown, mobilemanipulation device 900 is located in the room to pick up objects, toclean surfaces, to check on pets, to give potential thieves theimpression that someone is home, etc. Mobile manipulation device 900locomotes using wheels located in base 902. Base 902 additionallycomprises a computer for communicating with a remote control system,determining actuator commands from control information, providingactuator commands to mobile manipulation device actuators, providingcamera images to the control system, etc. Different views of the roomare available to mobile manipulation device 900 using top camera 904,arm camera 906, and wrist camera 908.

FIG. 10 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device. In some embodiments, the user interface ofFIG. 10 is provided to a user controlling a mobile manipulation device(e.g., via mobile manipulation device control system 102 of FIG. 1). Inthe example shown, the user interface of FIG. 10 comprises a cameraimage with an overlay comprising a plurality of command icons. Forexample, the camera image of FIG. 10 comprises an image captured by topcamera 904 of FIG. 9. The camera image of FIG. 10 shows mobilemanipulation device 1000. Mobile manipulation device 1000 is capable ofmoving laterally (e.g., left and right in the image of FIG. 10) androtating clockwise or counterclockwise. The overlay comprising aplurality of command icons comprises coarse overlay command icon 1002and fine overlay command icon 1004. Coarse overlay command icon 1002 andfine overlay command icon 1004 are associated with a motion in a thetaaxis of mobile manipulation device 1000, wherein the theta axis isassociated with a rotational movement of a base of mobile manipulationdevice 1000. In the example shown, coarse overlay command icon 1002 andfine overlay command icon 1004 are associated with a counterclockwisemotion in the theta axis, and the overlay additionally comprises acoarse overlay command icon 1014 and a fine overlay command icon 1016associated with a clockwise motion in the theta axis. Coarse overlaycommand icon 1002 and fine overlay command icon 1004 are associated withdifferent motion sizes, for example, coarse overlay command icon 1002 isassociated with a coarse motion (e.g., 10 degrees of rotation) and fineoverlay command icon 1004 is associated with a jog motion (e.g., 1degree of rotation). Coarse overlay command icon 1014 and fine overlaycommand icon 1016 are associated with different motion sizes, forexample, coarse overlay command icon 1014 is associated with a coarsemotion (e.g., 10 degrees of rotation) and fine overlay command icon 1016is associated with a jog motion (e.g., 1 degree of rotation). Otheroverlay icon functionality beyond coarse and fine jog motions can alsobe achieved.

The overlay comprising a plurality of command icons additionallycomprises coarse overlay command icon 1006 and fine overlay command icon1008. Coarse overlay command icon 1006 and fine overlay command icon1008 are associated with a motion in an y axis of mobile manipulationdevice 1000, wherein the y axis is associated with a horizontal movementof a base of mobile manipulation device 1000. In the example shown,coarse overlay command icon 1006 and fine overlay command icon 1008 areassociated with a rightward motion in the y axis, and the overlayadditionally comprises a coarse overlay command icon 1020 and a fineoverlay command icon 1018 associated with a leftward motion in the yaxis. Coarse overlay command icon 1006 and fine overlay command icon1008 are associated with different motion sizes, for example, coarseoverlay command icon 1006 is associated with a coarse motion (e.g., amovement of the base 10 cm to the right) and fine overlay command icon1008 is associated with a jog motion (e.g., a movement of the base 1 cmto the right). Coarse overlay command icon 1020 and fine overlay commandicon 1018 are associated with different motion sizes, for example,coarse overlay command icon 1020 is associated with a coarse motion(e.g., a movement of the base 10 cm to the left) and fine overlaycommand icon 1018 is associated with a jog motion (e.g., a movement ofthe base 1 cm to the left).

The overlay comprising a plurality of command icons additionallycomprises arm camera switch indication button 1010 and wrist cameraswitch indication button 1012. In the event an indication to arm cameraswitch indication button 1010 is received (e.g., by the mobilemanipulation device control system) a user interface including a cameraimage from an arm camera is shown. In the event an indication to wristcamera switch indication button 1012 is received (e.g., by the mobilemanipulation device control system) a user interface including a cameraimage from a wrist camera is shown. In some embodiments, the interfacedisplays more than one video stream at the same time (e.g., an interfacethat shows video from the top camera, arm camera, and wrist camera atthe same time), each with its own overlay.

In some embodiments, the overlay comprises a plurality of command iconsadditionally comprises a torque overlay for displaying an actuatortorque. For example, a command icon is colored (e.g., colored red) whenan associated actuator is delivering a torque above a threshold value, adisplay region is colored when an associated actuator is delivering atorque above a threshold value, the overlay comprises one or more torquemeter displays, etc. The visibility of the color can be proportional tothe torque, such as becoming less transparent with higher torques.

FIG. 11 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device. In some embodiments, the user interface ofFIG. 1 is provided to a user controlling a mobile manipulation device(e.g., via mobile manipulation device control system 102 of FIG. 1). Inthe example shown, the user interface of FIG. 11 comprises a cameraimage with an overlay comprising a command icon. For example, the cameraimage of FIG. 11 comprises an image captured by top camera 904 of FIG.9. In the example shown, the camera image of FIG. 11 shows mobilemanipulation device 1100. Mobile manipulation device 1100 includes arm1108 camera 1112 that points along arm 1108. Mobile manipulation device1100 has base 1110 that enables rotation of mobile manipulation device1100. The overlay comprising a command icon comprises fine overlaycommand icon 1102. Fine overlay command icon 1102 is associated with ajog motion in a counterclockwise direction in a theta axis. Fine overlaycommand icon 1102 comprises the only overlay command icon shown in theuser interface of FIG. 8. Fine overlay command icon 1102 comprises acursor icon (e.g., a mouse cursor icon or icon that appears when thevideo display is touched) comprising a shape indicating a jog motion ina counterclockwise direction in a theta axis. The motion axis,direction, and size associated with fine overlay command icon 1102 aredetermined using a cursor position or touch position. For example, thedisplay of FIG. 11 is divided into a set of regions, wherein each regionis associated with a different motion. The displayed cursor icon (e.g.,the cursor icon for the mouse cursor or icon that appears when the videodisplay is touched) is determined to be a cursor icon associated withthe region the cursor lies within. In some embodiments, multiple regionsrepresenting different sizes of motions (e.g., small, medium, and large)are used, or continuous variability of the motion size is commandedbased on the position of the mouse cursor or location of touch andindicated to the user via rendered computer graphics (e.g., variableicon size). The overlay comprising a command icon additionally comprisesarm camera switch indication button 1104 and wrist camera switchindication button 1106.

FIG. 12 is a diagram illustrating a set of command icon regions. In someembodiments, the diagram of FIG. 12 shows display regions associatedwith different command icons for an overlay command icon comprising acursor icon. In the example shown, each region of FIG. 12 is associatedwith a command icon that is displayed as a cursor icon when the cursoris in that region or a touch occurs in that region. When the cursor isnot in that region or the region is not being touched the command iconmay be entirely or partially hidden from the user. FIG. 12 comprises aplurality of regions. Region 1200 is associated with coarsecounterclockwise motion overlay command icon 1216 associated with acoarse counterclockwise motion in a theta axis. Region 1202 isassociated with fine counterclockwise motion overlay command icon 1218associated with a fine counterclockwise motion in a theta axis. Region1204 is associated with coarse clockwise motion overlay command icon1220 associated with a coarse clockwise motion in a theta axis. Region1206 is associated with fine clockwise motion overlay command icon 1222associated with a fine clockwise motion in a theta axis. Region 1208 isassociated with coarse rightward motion overlay command icon 1224associated with a coarse rightward motion in an y axis. Region 1210 isassociated with fine rightward motion overlay command icon 1226associated with a fine rightward motion in an y axis. Region 1212 isassociated with coarse leftward motion overlay command icon 1228associated with a coarse leftward motion in an y axis. Region 1230 isassociated with fine leftward motion overlay command icon 1200associated with a fine leftward motion in an y axis.

FIG. 13 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device. In some embodiments, the user interface ofFIG. 13 is provided to a user controlling a mobile manipulation device(e.g., via mobile manipulation device control system 102 of FIG. 1). Inthe example shown, the user interface of FIG. 13 comprises a cameraimage with an overlay comprising a plurality of command icons. Forexample, the camera image of FIG. 13 comprises an image captured by armcamera 906 of FIG. 9. In the example shown, the camera image of FIG. 13shows mobile manipulation device arm 1300. Mobile manipulation devicearm 1300 is capable of moving vertically and extending or retracting.The overlay includes a plurality of command icons—for example, coarsevertical up overlay command icon 1302 and fine vertical up overlaycommand icon 1304. Coarse vertical up overlay command icon 1302 and finevertical up overlay command icon 1304 are associated with a motion in az axis of a mobile manipulation device, wherein the z axis is associatedwith a vertical movement of a vertical lift of the mobile manipulationdevice. In the example shown, coarse vertical up overlay command icon1302 and fine vertical up overlay command icon 1304 are associated witha upwards motion in the z axis. Coarse vertical up overlay command icon1302 and fine vertical up overlay command icon 1304 are associated withdifferent motion sizes—for example, coarse vertical up overlay commandicon 1302 is associated with a coarse motion (e.g., 10 cm up) and finevertical up overlay command icon 1304 is associated with a jog motion(e.g., 1 cm up).

The overlay includes a plurality of command icons—for example, coarsevertical down overlay command icon 1314 and fine vertical down overlaycommand icon 1316. Coarse vertical down overlay command icon 1314 andfine vertical down overlay command icon 1316 are associated with amotion in a z axis of a mobile manipulation device, wherein the z axisis associated with a vertical movement of a vertical lift of the mobilemanipulation device. In the example shown, coarse vertical down overlaycommand icon 1314 and fine vertical down overlay command icon 1316 areassociated with a downwards motion in the z axis. Coarse vertical downoverlay command icon 1314 and fine vertical down overlay command icon1316 are associated with different motion sizes—for example, coarsevertical down overlay command icon 1314 is associated with a coarsemotion (e.g., 10 cm down) and fine vertical down overlay command icon1316 is associated with a jog motion (e.g., 1 cm down).

The overlay comprising a plurality of command icons additionallycomprises coarse retracting overlay command icon 1306 and fineretracting overlay command icon 1308. Coarse retracting overlay commandicon 1306 and fine retracting overlay command icon 1308 are associatedwith a motion in a x axis of the mobile manipulation device, wherein thex axis is associated with a retraction movement of an arm of the mobilemanipulation device. In the example shown, coarse retracting overlaycommand icon 1306 and fine retracting overlay command icon 1308 areassociated with a retracting motion in the x axis. Coarse retractingoverlay command icon 1306 and fine retracting overlay command icon 1308are associated with different motion sizes—for example, coarseretracting overlay command icon 1306 is associated with a coarseretracting motion (e.g., 10 cm) and fine retracting overlay command icon1308 is associated with a jog motion (e.g., 1 cm).

The overlay comprising a plurality of command icons additionallycomprises coarse extending overlay command icon 1318 and fine extendingoverlay command icon 1320. Coarse extending overlay command icon 1318and fine extending overlay command icon 1320 are associated with amotion in a x axis of the mobile manipulation device, wherein the x axisis associated with an extension movement of an arm of the mobilemanipulation device. In the example shown, coarse extension overlaycommand icon 1318 and fine extension overlay command icon 1320 areassociated with a retracting motion in the x axis. Coarse extensionoverlay command icon 1318 and fine extension overlay command icon 1320are associated with different motion sizes—for example, coarse extensionoverlay command icon 1318 is associated with a coarse extension motion(e.g., 10 cm) and fine extension overlay command icon 1320 is associatedwith a jog motion (e.g., 1 cm).

The overlay comprising a plurality of command icons additionallycomprises top camera switch indication button 1310 and wrist cameraswitch indication button 1312. In response to an indication to topcamera switch indication button 1310 being received (e.g., by the mobilemanipulation device control system), a user interface including a cameraimage from a top camera is shown. In response to an indication to wristcamera switch indication button 1312 being received (e.g., by the mobilemanipulation device control system), a user interface including a cameraimage from a wrist camera is shown.

FIG. 14 is a diagram illustrating an embodiment of a user interface fora mobile manipulation device. In some embodiments, the user interface ofFIG. 14 is provided to a user controlling a mobile manipulation device(e.g., via mobile manipulation device control system 102 of FIG. 1). Inthe example shown, the user interface of FIG. 14 comprises a cameraimage with an overlay comprising a plurality of command icons. Forexample, the camera image of FIG. 14 comprises an image captured bywrist camera 908 of FIG. 9. In the example shown, the camera image ofFIG. 14 shows mobile manipulation device manipulator 1400. Mobilemanipulation device manipulator 1400 is capable of moving in a pitchaxis, rotating in a roll axis, and opening or closing. Other types ofmanipulators can also be controlled using this type of controlinterface. For example, the manipulator may comprise a hook. The commandicons overlaying the camera image would be positioned and presented soas to enable the operator to efficiently operate the hook to pull open adrawer, for example. The overlay comprising a plurality of command iconsincludes coarse manipulator counterclockwise rotation overlay commandicon 1402 and fine manipulator counterclockwise rotation overlay commandicon 1404. Coarse manipulator counterclockwise rotation overlay commandicon 1402 and fine manipulator counterclockwise rotation overlay commandicon 1404 are associated with a motion in a roll axis of the mobilemanipulation device, wherein the roll axis is associated with a rotationof a wrist rotational coupling of the mobile manipulation device. Coarsemanipulator counterclockwise rotation overlay command icon 1402 and finemanipulator counterclockwise rotation overlay command icon 1404 areassociated with different motion sizes—for example, coarse manipulatorcounterclockwise rotation overlay command icon 1402 is associated with acoarse motion (e.g., 10 degrees) and fine manipulator counterclockwiserotation overlay command icon 1404 is associated with a jog motion(e.g., 1 degree).

The overlay comprising a plurality of command icons includes coarsemanipulator clockwise rotation overlay command icon 1418 and finemanipulator clockwise rotation overlay command icon 1420. Coarsemanipulator clockwise rotation overlay command icon 1418 and finemanipulator clockwise rotation overlay command icon 1420 are associatedwith a motion in a roll axis of the mobile manipulation device, whereinthe roll axis is associated with a rotation of a wrist rotationalcoupling of the mobile manipulation device. Coarse manipulator clockwiserotation overlay command icon 1418 and fine manipulator clockwiserotation overlay command icon 1420 are associated with different motionsizes—for example, coarse manipulator clockwise rotation overlay commandicon 1418 is associated with a coarse motion (e.g., 10 degrees) and finemanipulator clockwise rotation overlay command icon 1420 is associatedwith a jog motion (e.g., 1 degree).

The overlay comprising a plurality of command icons additionallycomprises coarse open overlay command icon 1406 and fine open overlaycommand icon 1408. Coarse open overlay command icon 1406 and fine openoverlay command icon 1408 are associated with an open/close motion ofthe mobile manipulation device, wherein the open/close motion isassociated with an opening and closing of manipulator 1400 of the mobilemanipulation device. In the example shown, coarse open overlay commandicon 1406 and fine open overlay command icon 1408 are associated with anopening motion. Coarse open overlay command icon 1406 and fine openoverlay command icon 1408 are associated with different motion sizes—forexample, coarse open overlay command icon 1406 is associated with acoarse motion and fine open overlay command icon 1408 is associated witha jog motion.

The overlay comprising a plurality of command icons additionallycomprises coarse closed overlay command icon 1422 and fine closedoverlay command icon 1424. Coarse closed overlay command icon 1422 andfine closed overlay command icon 1424 are associated with an open/closemotion of the mobile manipulation device, wherein the open/close motionis associated with an opening and closing of manipulator 1400 of themobile manipulation device. In the example shown, coarse closed overlaycommand icon 1422 and fine closed overlay command icon 1424 areassociated with an opening motion. Coarse closed overlay command icon1422 and fine closed overlay command icon 1424 are associated withdifferent motion sizes—for example, coarse closed overlay command icon1422 is associated with a coarse motion and fine open overlay commandicon 1424 is associated with a jog motion.

The overlay comprising a plurality of command icons additionallycomprises coarse vertical up overlay command icon 1410 and fine verticalup overlay command icon 1412. Coarse vertical up overlay command icon1410 and fine vertical up overlay command icon 1412 are associated witha motion in a pitch axis of the mobile manipulation device, wherein thepitch axis is associated with a bending of a wrist hinge of the mobilemanipulation device. In the example shown, coarse vertical up overlaycommand icon 1410 and fine vertical up overlay command icon 1412 areassociated with an upwards motion in the pitch axis. Coarse vertical upoverlay command icon 1410 and fine vertical up overlay command icon 1412are associated with different motion sizes—for example, coarse verticalup overlay command icon 1410 is associated with a coarse motion (e.g.,10 cm) and fine vertical up overlay command icon 1412 is associated witha jog motion (e.g., 1 cm).

The overlay comprising a plurality of command icons additionallycomprises coarse vertical down overlay command icon 1426 and finevertical down overlay command icon 1428. Coarse vertical down overlaycommand icon 1426 and fine vertical down overlay command icon 1428 areassociated with a motion in a pitch axis of the mobile manipulationdevice, wherein the pitch axis is associated with a bending of a wristhinge of the mobile manipulation device. In the example shown, coarsevertical down overlay command icon 1426 and fine vertical down overlaycommand icon 1428 are associated with an downwards motion in the pitchaxis. Coarse vertical down overlay command icon 1426 and fine verticaldown overlay command icon 1428 are associated with different motionsizes—for example, coarse vertical down overlay command icon 1426 isassociated with a coarse motion (e.g., 10 cm down) and fine verticaldown overlay command icon 1428 is associated with a jog motion (e.g., 1cm up).

The overlay comprising a plurality of command icons additionallycomprises arm camera switch indication button 1414 and top camera switchindication button 1416. In the event an indication to arm camera switchindication button 1414 is received (e.g., by the mobile manipulationdevice control system) a user interface including a camera image from anarm camera is shown. In the event an indication to top camera switchindication button 1416 is received (e.g., by the mobile manipulationdevice control system) a user interface including a camera image from atop camera is shown.

FIG. 15 is a flow diagram illustrating an embodiment of a process for adisplay window control system for a mobile manipulation device. In someembodiments, the process of FIG. 15 is executed by mobile manipulationdevice control system 102 of FIG. 1. In the example shown, in 1500, animage is received from a mobile manipulation device. For example, animage or a plurality of images are received from one or more of a topcamera, an arm camera, a wrist camera, etc. In 1502, a display output todisplay the image in a display window is determined. For example,determining a display output comprises determining a selected camera orset of cameras, determining images associated with selected cameras,cropping images, zooming images enhancing images, rotating images,combining images, etc. In 1504, an overlay output to display a coarseoverlay command icon or a fine overlay command icon is determined. Othertypes of overlay command icons can be supported besides coarse and fineoverlay command icons. The overlay command icons can be placed in thedisplay at prespecified locations that are intuitively mapped to theirfunction (e.g., an upward motion icon appears at the top of the screen).The overlay command icons can also be placed in the display at locationsthat are automatically generated based on the current state of the robot(positions of its links in the image for example). The coarse overlaycommand icon or the fine overlay command icon comprise command icons forindicating a mobile manipulation device command. For example, onecommand icon is determined, one coarse overlay command icon and one fineoverlay command icon is determined, or a plurality of coarse overlaycommand icons and fine overlay command icons are determined, eachcommand icon associated with a motion axis and a direction. In 1506, thedisplay output and the overlay output are displayed. For example, thedisplay output is displayed with the overlay output as an overlay on adisplay of a mobile manipulation device control system. In 1508, it isdetermined that an indication has been received to activate the coarseoverlay command icon or the fine overlay command icon. In 1510, theindication is provided to the mobile manipulation device.

FIG. 16 is a flow diagram illustrating an embodiment of a process fordetermining a display output to display the image in a display window.In some embodiments, the process of FIG. 16 implements 1502 of FIG. 15.In the example shown, in 1600, a camera selected for display isdetermined. In 1602, an image associated with the selected camera isdetermined. In 1604, the image is processed. For example, processing theimage may comprise scaling the image, translating the image, rotatingthe image, dewarping the image, cropping the image, or any otherappropriate image processing. In 1606, it is determined whether a switchindication is received to switch the display output. In the event it isdetermined that a switch indication is received, control passes to 1600.In the event it is determined that a switch indication is not received,the process ends.

FIG. 17 is a flow diagram illustrating an embodiment of a process fordetermining an overlay output to display a coarse overlay command iconor a fine overlay command icon. In some embodiments, the process of FIG.17 implements 1504 of FIG. 15. In the example shown, in 1700, a cameraselected for display is determined. In 1702, one or more axes associatedwith the camera selected for display is/are determined. For example, any axis and a theta axis are associated with a top camera, a x axis and az axis are associated with an arm camera, and a pitch axis, a roll axis,and an open/close motion are associated with a wrist camera. In 1704, anext axis associated with the camera selected for display is selected.In some embodiments, the first axis associated with the camera selectedfor display is selected. In 1706, a coarse overlay command icon or afine overlay command icon associated with the axis are determined for afirst direction. In 1708, a coarse overlay command icon or a fineoverlay command icon associated with the axis are determined for asecond direction. For example, determining a command icon comprisesdetermining a command icon shape, determining a command icon size,determining a command icon orientation, determining a command iconlocation, etc. In 1710, it is determined whether there are more axes.For example, it is determined whether there are more axes associatedwith the camera selected for display. In the event it is determined thatthere are more axes, control passes to 1704. In the event it isdetermined that there are not more axes, control passes to 1712. In1712, one or more switch indication icons are determined, each switchindication icon associated with another camera.

FIG. 18 is a flow diagram illustrating an embodiment of a process fordetermining an overlay output to display a coarse overlay command iconor a fine overlay command icon. In some embodiments, the process of FIG.18 implements 1508 of FIG. 15. In the example shown, in 1800, a cameraselected for display is determined. In 1802, a set of control regionsassociated with the camera selected for display is determined. In 1804,a current cursor position or touch position is determined. In 1806, acontrol region of a set of control regions associated with the currentcursor position or touch position is determined. In 1808, an overlaycommand icon associated with the control region in the current cursorposition or touch position is determined. In 1810, one or more switchindication icons are determined, each switch indication icon associatedwith another camera.

FIG. 19 is a flow diagram illustrating an embodiment of a process for aremote control interface system for a mobile manipulation device. Insome embodiments, the process of FIG. 19 is executed by mobilemanipulation device 104 of FIG. 1. In the example shown, in 1900, animage from a mobile manipulation device camera is provided. In 1902, anindication to move is received. For example, the indication to movecomprises an indication received to activate a coarse overlay commandicon or a fine overlay command icon. In 1904, actuator commands aredetermined based at least in part on the indication to move. In 1906,the actuator commands are provided to one or more actuators.

FIG. 20 is a flow diagram illustrating an embodiment of a process fordetermining actuator commands based at least in part on an indication tomove. In some embodiments, the process of FIG. 20 implements 1904 ofFIG. 19. In the example shown, in 2000, an axis associated with theindication to move is determined. In 2002, a motion size associated withthe indication to move is determined. In 2004, a motion directionassociated with the indication to move is determined. In 2006, one ormore actuators associated with the axis are determined. In 2008,actuator commands for the one or more actuators are determined based atleast in part on the motion size and the motion direction.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A display window control system for a mobilemanipulation device, comprising: an interface configured to: receive animage from a mobile manipulation device; and a processor configured to:determine a display output to display the image in a display window;determine an overlay output to display one or more overlay commandicons; determine that an indication has been received to activate anoverlay command; and provide the indication to the mobile manipulationdevice.
 2. The system of claim 1, wherein the image is received from anoverhead camera of the mobile manipulation device, an arm camera of themobile manipulation device, or a wrist camera of the mobile manipulationdevice.
 3. The system of claim 1, wherein the one or more overlaycommand icons comprise a coarse overlay command icon and a fine overlaycommand icon.
 4. The system of claim 1, wherein the overlay output isdetermined based at least in part on a camera associated with the image.5. The system of claim 3, wherein the coarse overlay command icon isassociated with a coarse jog motion, and wherein the fine overlaycommand icon is associated with a fine jog motion.
 6. The system ofclaim 1, wherein the processor is further configured to determine that aswitch indication has been received to switch the display output todisplay a second image associated with an indicated camera of the mobilemanipulation device.
 7. The system of claim 1, wherein an axisassociated with the overlay command icon is based at least in part on acamera associated with the display output.
 8. The system of claim 3,wherein the coarse overlay command icon or the fine overlay command iconare associated with a motion in an y axis of the mobile manipulationdevice, wherein the y axis is associated with a horizontal movement of abase of the mobile manipulation device.
 9. The system of claim 3,wherein the coarse overlay command icon or the fine overlay command iconare associated with a motion in a theta axis of the mobile manipulationdevice, wherein the theta axis is associated with a rotational movementof a base of the mobile manipulation device.
 10. The system of claim 3,wherein the coarse overlay command icon or the fine overlay command iconare associated with a motion in a z axis of the mobile manipulationdevice, wherein the z axis is associated with a vertical movement of avertical lift of the mobile manipulation device.
 11. The system of claim3, wherein the coarse overlay command icon or the fine overlay commandicon are associated with a motion in a y axis of the mobile manipulationdevice, wherein they axis is associated with an extension movement of anarm of the mobile manipulation device.
 12. The system of claim 3,wherein the coarse overlay command icon or the fine overlay command iconare associated with a motion in a pitch axis of the mobile manipulationdevice, wherein the pitch axis is associated with a bending of a wristhinge of the mobile manipulation device.
 13. The system of claim 3,wherein the coarse overlay command icon or the fine overlay command iconare associated with a motion in a roll axis of the mobile manipulationdevice, wherein the roll axis is associated with a rotation of a wristrotational coupling of the mobile manipulation device.
 14. The system ofclaim 3, wherein the coarse overlay command icon or the fine overlaycommand icon are associated with an open/close motion of device gripper.15. The system of claim 1, wherein the overlay output additionallycomprises a torque overlay for displaying an actuator torque.
 16. Aremote control interface system for a mobile manipulation device,comprising: an interface configured to: provide an image from a mobilemanipulation device camera; and receive an indication to move; and aprocessor configured to: determine actuator commands based at least inpart on the indication to move; and provide the actuator commands to oneor more actuators.
 17. The system of claim 16, wherein the indication tomove is associated with a command motion type.
 18. The system of claim17, wherein the command motion type comprises a coarse motion, a finejog motion, or an autonomous motion than concludes within a boundedamount of time.
 19. The system of claim 16, wherein the indication tomove is associated with a command axis, wherein the command axiscomprises an x axis, a theta axis, a z axis, a y axis, a pitch axis, aroll axis, or an open/close motion.